What was changed

• Added scripts/run_bench.py and a GH workflow that runs it nightly or can be manually triggered
• Updated README to clarify the cost of third-party module imports
• Fixed issue for workflows defined in __main__ module

Results

(if not wanting horizontal scrollbars on table, open dev console and remove overflow CSS property for .markdown-body table class...I don't feel like reworking the tables)

Sandboxed

Unsandboxed

Notes

Notes:

• The workflow tested is a simple workflow that accepts a string, invokes an activity w/ said string, and relays back the activity's response as its own
• The linux runner is our 4-core org-level one and the windows runner is the GH-provided one
• max_concurrent up there applies to both max_concurrent_workflow_tasks and max_concurrent_activities (set as the same number for now)
• Due to the nature of Python, single-worker/process benchmarks won't show the true power of the system. Python is inherently single-threaded on CPU-bound tasks such as these. It is entirely likely performance scales up linearly proportional to worker process count.
• Much of the larger tests fought Temporalite for resources.
• Note how the workflows-per-second number varies wildly on some scenarios. This may be a product of Temporalite running alongside and its performance unpredictability.
• Lots of linux warnings and errors happen when stopping Temporalite via the Rust ephemeral server shutdown. We are probably not doing this right.
• Adding an import for even a single third party library tripled or more the memory usage. This is because each import is reloaded and isolated per workflow. README updated to discourage workflows from importing non-passthrough, non-standard-library modules from the same file the workflow is defined in. This won't show up much for small workflow counts.
• The workflows-per-second times above include the amount of time taken for a client waiting on its response to fetch from server (and convert the response, etc).
• By default, max_cached_workflows is 1000, max_concurrent_workflow_tasks is 100, and max_concurrent_activities is 100. In light of these numbers above, should we increase those? Also note wrt activities it affects sync activities too.

Things that could be added but weren't:

• Measurement of CPU
• Multiple workers in separate processes
• Better result output for trending

The goal of this project was to just ensure the SDK was good enough performance wise. We'll need to spend more time optimizing.

Checklist

1. Closes Stress tests #23